Summary

This PR adds comprehensive Kubernetes deployment support for llm-katan, enabling multi-instance deployments with model aliasing capabilities.

Kubernetes Manifests (Kustomize-based)

• Base deployment with security contexts and health probes
• PersistentVolumeClaim (5Gi) for efficient model caching
• Service (ClusterIP) exposing port 8000
• Namespace isolation (llm-katan-system)

Multi-Instance Support (Overlays)

• gpt35 overlay: Serves gpt-3.5-turbo alias
• claude overlay: Serves claude-3-haiku-20240307 alias
• Isolated PVCs per instance (prevents ReadWriteOnce conflicts)
• Common labels component for consistent resource labeling

Model Caching Optimization

• InitContainer (model-downloader) pre-downloads models to PVC
• Smart caching: Skips download if model exists
• Uses python:3.11-slim + hf download for ~45MB lightweight init
• Main container starts instantly with cached model

Bug Fix (config.py)

• Added YLLM_SERVED_MODEL_NAME environment variable support
• Previously only worked via CLI arguments
• Now enables Kubernetes env-based configuration

Documentation

• Comprehensive deployment guide (deploy/docs/README.md)
• Architecture explanation (Pod structure, storage, networking)
• Kind cluster setup examples
• Troubleshooting section with common issues

Test Results

Deployment Validation (Kind Cluster)

Resources Created:

• Namespace: llm-katan-system
• Deployments: llm-katan-gpt35, llm-katan-claude (both 1/1 Running)
• Services: llm-katan-gpt35, llm-katan-claude (ClusterIP, port 8000)
• PVCs: llm-katan-models-gpt35, llm-katan-models-claude (both 5Gi Bound)

API Validation:

• GPT35 instance
  $ curl http://llm-katan-gpt35:8000/v1/models
  {"data":[{"id":"gpt-3.5-turbo",...}]}

• Claude instance
  $ curl http://llm-katan-claude:8000/v1/models
  {"data":[{"id":"claude-3-haiku-20240307",...}]}

Motivation

This implementation addresses the need for:

• Cloud-native deployments: Production-ready Kubernetes manifests
• Multi-instance testing: Run multiple model aliases simultaneously
• Efficient resource usage: Model caching prevents redundant downloads
• Testing flexibility: Easy overlay creation for new model aliases

The Kustomize structure enables:

• Consistent base configuration
• Environment-specific customization via overlays
• Easy addition of new model aliases without base changes

Related issue: #278